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Chapter I The Prob I em 


Cons ifier the probi am of proafamm I no a computer to 
recognize objects with smoothly curved surfaces, such as 
the object [A the- photopraph *f figure 1.1* hsfies such 
as these can be digitized by an tins ge-ri I SSflC tor Carers,, 
so that the picture ts represented by a raster of 
Intensities at closely Spaced sample points, represented 
numerically In figure 1.2 + We will cons fder a method 0* 
prqgess i ng such input with the ultimate goal of 
recognizing the object th the Image. 

There are numerous more or less adequate known 
techniques tor classifying an image once significant 
features have beer, extracted from ft, but the problem of 
extracting such features from the basic optical data is 
less well uncerstood* The methods which will be 
discussed here arc 11 1 o w-l eve I \ In that they manipulate 
actual picture points and try 10 extract salient 
features, rather than wor k! n o with high-level 
descriptions and attempting to produce an tdent i f icet ton, 
ft must be recognized, however, That The SO- 
called high- and low-level aspects Of vision cannot 
realty be cleanly separated. There E s no foolprdO* 


Fifti.ir<2 L*lr A Simple SmoDthly Curved Object 






Figure l-St Sampled Light Intensities from the Apple 

ei figure 1.1 


The intensities in this array have heeri scaled to be between 

0 and 99 
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com p I ete I y local way to find f eatures, a S there */l t t 
always be ambiguities Which can only be resolved through 
the use of context, For example, one must know the light 
Intensity fat least roughly) En order to deter mine 
whether an object is white or black, as & white object in 
very dim light can easily reflect less light than a black 
object In sunlight* A plum cannot be easily 
distinguished from an Isolated grape, unless the slie Is 
known* H Ig h I ights oil a sn doth surface Cannot be 
under stood unless the form of the Illumination is known. 
The context can ot course be determined partly 
from the scene Itself, For example, a real scene will 
generally contain surfaces with a w Ido range of 
reflectivities. This establishes a liebt Intensity 
frame of reference Tn which the lighter objects will 
appear white and the darker ones black. One cannot teM 
the s 17 a of a white sphere alone En a photograph, but ft 
It is shpwn next to a tennis b g I I , its 5 1 j e Is known by 
comparison* (It Is possible, but unlikely, that the 
tennis ball 1$ actually a scaled-up model three feet In 
diameter, ThTs usually happens only on movie sets.) In 
a similar manner, the highlight on g known object gives 
Information about the lighting which can be used to 
Interpret the highlights on Other Objects fn the Image, 


So far, the u SO of context has been considered 
only on the level of object identification. Actually, 
Context fs even more necessary at the level df finding 
Visual parts Of objects, such as cO efts, h line-finding 
program can be saved an enormous amount of work Tf It Is 
told approximately where to look. If S program thinks it 
Is seeing an apple. It can know that a good way to ver Efy 
this hypothesis is to look on top for a stem, 

A program can only make use of these Cues, 
however. If it can pass information resulting from a 
partial identification back +o the low-level feature- 
find Trig rout Tna?. This sort of system shal l be referred 
to as "vert ice r ", in the sense that control passes 
frequently between h T gh« anfi low—fever routines. The 
term "hor I ?onta J " refers to a syst&m which works In 
Stages, each of which produces a more abstract 
representation of the scene. Much of the previous work 
Fn vision has been of this sort, A typical sequence 
might be to remove noise, enhance features, extract 
features, group them, and then identify objects* Since 
no provision is made fn e hor r Eon fa I system for passing 
Information back down this chain, the system cannot make 
use of context Information obtained from the imago 


The not hod 5 which will bo presented ho r e are 


intended To fit into a vertical system in "wq ways. 

Firsts they can be ..-sed to st&rt oft ?- vertical system 
with Enfcirniatl-On pood enough to get it going, Sec□ rid j 
they extract features which are useful 1 for object 
tdertif [cation. These features will be extracted in 
such a Tian.net A S tc allow easy advantage to be derTved 
from contort Information, 

This work is intended to be a step towards making 
computers see* This goal is Interesting for a number of 
reasons. Computers with vision would be useful for 
applications In au fomSf ion, and wou d be able +0 Interact 
bettor with humans, Corrpu'rr vi-ion may weir provide 
Instructive models tor the understanding of hunan vision. 
The problem is also very Interesting Tn Its own rl gh t f as 
an aspect of the study of Artificial Intel! Igence, 


Chapter 7 . Previous Work 


Techniques have been Invest fan ted which could be 
applied to s noo t h I y curved objects as a Step towards 
recogh ft Ign* 

2 * I Shape from Shading 

It Is possible to f f nd 5 groat deal about the 
shape of a smoothly curved object from a single monocular 
image, given a knowledge D f [ts Surface reflection 
properties arid the posrtron and nature Of the light 
Sources. Horn [lQ^] generates curves lying on the 

surface of the object by an Iterative solution of a set 
Of differential equations relating shape to the intensity 
of imaee po 1 nt 5 „ Similar methods have been app] ied to 
the analysis of lunar topography from Lunar Orblter 
photographs Cl 4,3]* 

Th[& method requires a uniform object surface. 

Its reflectance must be a smooth function of the angle 
the surface makes wTtti the Incident and exit rays, Any 
marks on the surface will disrupt the solutions to the 
differential equations, although very small marks can be 
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ignored by s p p r op r f a t e averaging tdchn iquss, 

Sfn,ce his results depend 50 heavily upon the 
Object surface properties, Hem [l(Q has undertaken an 
investigation of how typical surfaces behave. To within 
a reasonable approximation, the ref I ecta no b of b real 
object can be considered to be a simple I In ear 
Superposition of matte and specular components. The 
specular component refers to the light reflected such 
that the angle of incidence equals the angle of 
reflection, as If the surface were mirrored. The matte 
component Is light which Is scattered by the surface* 

The simplest model of & matte surface Is b n f ambert la n 11 
surfacB, which emits light uniformly In all directions, 
regaro i ess of the incident angle. The physics I effects 
which produce these two components are different, as 
shown by the fact that they frequently have different 
spectral properties. The specular component has the 
spectral distribution of the Incident light, while the 
matte component, which penetrates the surface more deeply 
before being scattered, is spectrally distributed as the 
product of the Incident light spectrum and the spectrum 
of the object pigment* 


2.2 Detect ion of Optical Ed g es 

M y c h research has gore into the d £ t £C t i C h & n d 
tracing of contrast edges Fn an image, T he se edges can 
be emphasized by ditfseentia11 on preprocessing 
operations^ such as the gradient or tap lac fan , 

?,?*! P lanc-su rf aced Objects 

Edge detection Is particularly attract Eve for 
plane surfaced objec+s* Since the edges are straight 
lines Ethe intersection of twd planes), a determination 
of the position Of the edges completely specifies the 
position ot the plane surface which they enclose, and an 
odgs itself can he located In terms of just a few of Its 
po f nt s, 

A program by L. 6* Roberts recognizes white plane 
surfaced objects on a dark background I &!]* He considers 
objects which can be put together out of g jet of g Wen 
sub-shapes,, such as rectangular parallelepipeds and 
wedges* The image is first d 1 f f erent ia t ed „ Lines are 
then found In the resulting picture by a mu l + t p le^step 
procedure^ first fitting short lines to Iccsl areas* 
ct 1 ffl p na+ 1 ng t r n y loop?* then fitting longer and longer 


lines lo the s h e r T c r oiifis, end finally no n e r a + I ”i 0 a 
I east-mean-sou are I ime which is taken, to represent the 
o r t p f n e I e d r; e . 

The next phase ts recoen It ion of polygons In the 
line drawing! followed by the matching of sets of 
p-o Pygons against the poss Ibl e mc-de I s. T he m a t c h i n a is 
first done on g straight to pog ra p h 1 0 3 i basis. The two- 
dimensional projection Of a brick, for instance* 
generally contains three auedr1 I a ter□Is with one corner 
point In comma n, Jtp such point exists on a wedge. 
Assuming, then, that this point corresponds to the corner 
of a brick, the program can match the other iines and 
points in the quadrilaterals to what must then be the 
corrospond ire lines and points of the model, A least- 
meen-square error matrix procedure is then ysed to find 
the best brick tin 3-spaceS which generates the given 
Two-dimensional line drawing, H the lea st-mea n - s oua r e 
error is small enough, the fit is accepted as correct. 

When g set of lines are matched by a model, the 
model can then be projected back onto the line drawing, 
but now with a I I of the hidden lines present. The mo d a 1 
Is new "removed 11 from the \ l nc drawing* which may entail 
the deletion of same lines, bet also may entail the 
addition Of some others. The procedure is now Iterated 


n 


until a I 1 of the lines of the input figure S 15 b ee n 
accounted for. Thus objects are recognized as ba ina 
Compounded of a number of the basic building blocks* 

Roberts depends on a h 1 0 h deer ee of precision of 
ff| e£i £u r eitifl nt Cf the position Of the edges, Since lie uSeS 
perspective In an essential way* Unfortunately, his 
procedure Fs useless for objects lacking straight IFne 
edges* One particularly Interesting aspect of Roberts ' 1 
work is his u sf of a poverty! interra! model of the 
potential object In the Image. A. similar approach might 
be useful for scenes consisting of regular Smoothly 
curved objects such as spheres and cylinders,, but it is 
difficult to envision Successful results using more 
amorphous forms. 

A pro-gram by R. W, Gosper visually locates white 
rectangular parallelepipeds on a black table* Due to the 
h 1 gh reflectance difference between the objects end the 
background, the outer edges are very clearly d e f 1 n ed * 

EThe program also finds Interior edges of the object 
where the contrast between adjacent fsces is high 
enough*! 1 he edges are found by an algorithm which scans 
In 3 line perpendicular to the edge, end mov e S this line 
along the edge from one end to the other* From the 
position of the edges In the Image, and the knowledge 


that "the object Is ractsnnu lar and rests on a table whose 
position is known, the exact throe-space position of the 
object can be calculated from Its hexspcnal outline. 

Griffith fal has made a study of the edges of 
geomatr Its-1 objects, and has developed 3 theory of their 
optical detection In the presence of noise. HEs system 
is designed to be easily integrated Into a vertical 
v r s Ion system, and Includes a high-level line proposer 
and verIfler . 

Guzman [ "o arses'’ a Straight-line drawing Into 
[ts component pi ana-surfaced objects. His work is 
notable here In that |t depends Mttle upon accuracy of 
measurement, and Is concerned with dissecting a complex 
stone Into Individual objects prior to de f ermin Ing their 
exact position or shape. In these respects It has goals 
similar to those of this work with respect to smoothly 
curved objects, Guzman's technleues could in fact be 
extended to smoothly curved Objects, and used to 
complement the methods discussed In the- next chapter; his 
methods do not depend so much upon the edges being 
straight, as do Roberts and Gasper. 

Different recognition methods ca 1 i for different 
degrees of precision In the final determination of line 
positron, Roberts requires high precision, because of 


hit use of second-ordEr perspective effects* The u se of 
stereo distance determination would alto recirlre suth 
high pr cc is Fon . GoSper requires only rrc d I uni precision. 
His go e I is to actually pick up the block, which on I y 
requires locating It to Kith in a centimeter or SO. hlo 
perspective,. sfereo, or other second-order effects are 
used I 50 the calculated position is not as sensitive to 
small errors In the line position. The prcaroms of 
GuEman and Griffith require only low precision, except Tn 
a few parts which make use of the pare! let Ism of i wo 
F Fnes, 

?, Z * 2 Curved Edges 

There has been much study of recognition of 
alphanumeric characters, Black characters on a white 
background provide h 1 gh-c on tra st edpeg* and some 
character-recognition programs work by tracing around the 
Character's edge. There has beeh M fH e ed ge -o r I e nt ed 
research on images derived from tfirco-dImonS Iona 1 
objects, and the result 5 of the two-d Imens Iona 1 work has 
little relevance to this problem* 

It is considerably easier to find 3 straight edge 
than a curved one, since only two points determine a 
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straight line, and additional points can then be verified 
by very sensitive test s* It many tests ere positive 
along a straight lire, the ex f st ence of the edge can then 
be asserted with a high statistical: con f i den e a, as by 
G r E t f I t h 1 s prograns* These t ec h n i qe CS can be used only 
over a short Interval ter a curved edge* 

2,5 The ir R Batons ” A p p r CsC h 

instead of looking for hIch^eontrast edges, sons 
pattern reoogin f t ion methods Took for homogeneous, areas of 
low contrast. Analysis then proceeds from the shape and 
Intereletions between these "regions"* There ere a 
number of tcctin F ques for characterizing the shape of a 
region, such as various moments or more complicated 

Shape descriptors C3j * Kitsch [lij analyzes 
photomicrographs of cells by hu Siding a tree structure of 
image regions with various levels of homogeneity* Mis 
methods ore the closest In the literature to those which 
are developed fn this theses, 


2+4 T extur^ I 1 Information 


The optical behavior o t an object depends very 
much on the texture of Its surface. The word "tenure" 
nav refer to either markings or departures from a smooth 
suMnce, but In e rth-er case they must be small compered 
with the size of the object In order to be considered 
texture, Texure analysis may be done by a w r de variety 
of methods, such as Tourter analysis or cross- 
correlation, Tex lure has been used +0 advantage in a 
range of studies, in such areas as recognition of terrain 
types CffrJ or cell Images Cl3} + Different types of 
texture will be discussed further In section 3.9. 
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Figure 3.1; The Incensity"regian Tree 


S(t> (white points) Threshold 


The Tree Structure 














Chapter 3 ftepre sent log an Image as an l nt ens ity-^region 
Tree 


3* I The Baste Method Used 


Cons td er an image, I, defined flfl a rtcfafiOu I ftf 


res+er of points, so that l!p) Is the light intensity at 
the point p, for any given I Ight Intensity threshold t, 



set of points 5 E t) ■= /o 


define 5 


the set of 


poFnts of rntensity t or greater, tech of the eight 
pictures In figure 3,1 (previous page) shows such a set 
of po 1 n+s, for some threshold* For any t * the set S (T ) 
can be partitioned into disjoint connected subsets Rjft}, 
Vh tch will henceforth he Called Pl regions 111 . Thus: 


s<t> ■ s,(+>( jr !L {+jU‘ “ -U R h* + >i 



If l^j, and each R; Is a connected set 


of points. Note th a r MtjjCZstt, ) if t^t, „ so each 
region at threshold t^ must be a subset o* some region at 
t f . The regions thus fall naturally into □ tree 
structure basc-dcn this subset relation, as shown In 
f i gu r a 3.1* 


Another particularly graphic way of looking at 
the tree Is to visualize the intensity function plotted 
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in the- form i j» f ( x r y ) 4 S I r c T n g this function with a 
horizontal plan© at severs! threshold levels, the tree 
Can be pictured hs irs figure 3,2* An Intensity Contour 
map of the pear Is shown In figure 3.3 In order to shew 
hew the r eg fens are actual i y nested, 

3*2 Cua nt I sat I on 

Choosing a set of threshold levels -^t ■ | Is 
eoulvaPent to quantizing the fight Intensities In the 
InagSj, In terms of the information retained in the tree. 
The more threshold levels In the sat, the greater the 
depth of the tree generated u 5 i n e these levels, We will 
generally consider threshold sots which are evenly spaced 
In the log of the light Intensity^ although a tree could 
be generated from any arbitrary sot of levels* Using the 
log of the light intensity generates a tree whose 
structure remains basically the seme rt the Illumination 
Is scaled up or down by a constant factor, 

3*3 Geometry of the Tree 

In the limit cf a continuous tree (In which the 
spacing bat wee n threshold levels approaches zero), the 
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Figure 3,2i The Pianos Shown S* Slices o£ the intensity 

F u nc C ki n 


J-ight Intensity 
2 = f(*py) 

























Figure 3-,3-j A Contour Map of the £ear 
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tips of the branches, represent local maxima in the Inace, 
beginning st a branch T | p and moving along It in the 
C ir ect Fon of I pw er intensity, t he r eg ion ex sands froni Thft 
max Frr.ym point to incFude other nearby points, essum-lng 
the Intensity function Is continuous in that area. Each 
treo branch can thus be t h q u g ht of as a growing region, 

A fork In the tree occurs whenever two or norC of these 
regions combine, terming one new larger region. In this 
case, the branch associated with the sub-region of 
largest area shall bo considered the "main branch* 1 , and 
the Other branches shall becnPled "sub-branches' 1 . It 
tho original Image is slightly no1sv, then as a region 
1r expands" (moving eFong e tree branch from hloh to iow 
intensity), ft win engulf large numbers of smaller 
regions whFch appear ahead of Its advancing edge, 
resulting |n frany short sub-branches on the tree. When 
two regions Of substantial area are combined. It is not 
really Important which Is considered the sub-branch. 

The highest region on the tree represents the 
brightest point in the Image, If the threshold Is 
lowered far enough, all ot the regions will eventually 
merge into one region containing all of the image points, 
this shall be referred to as the "root" 1 af the tree, 


Trees with Inconplete Region Information 

In the proceeding d r sc u s S i cn , + h e re-qions 
themselves have been considered to fee the elements of 1 ho 
tree- Let US no* consider so abstract tree structure fn 
which the elements of the tree are not the regions 
themselves,, bu 1 nodes containing i hf or I* a t i o n about these 
regions. Such a tree shall be colled an ,p liflSfle Tree 11 , 

1 f each node contains a complete C eser [ pt ' or, of the 
region to which it corre-spond& £that is. If RjCtjl Is 
given tor all i and +j), then the tree contains enough 
data to fee able to re-construct the image exact] y, to 
within the Units imposed by the quantization* 

[f each node contains only statistics of the 
corresponding regifirtj, r&ther than Fi complete description 
of the region, then the tree contains leas information 
than the original Image* These art the interesting 
trees, despite the fact that the image cannot be 
reconstructed from then. The problem of pattern 
recognition can be viewed as one of throwing axay 
Information In □ selective wav* To go from a picture of 
an apple to the word M appie N represents an enormous 
reduction in information ( ,f a picture Is worth a thousand 


word s") 


In general „ the nccfcs may contain any arbitrary 
set Of functions Of the corresponding region* In 
particular* the ones which will be used arc the position 
of the region's center of mass ix ft y t } t the area A of the 
region ti,e. the number of points In 1+3% antr a measure 
of the second moment about the Center of iflass, called the 
accen + r fc | + y e* 

The eccentricity Ts defined by 



all p t s p 
In rag i oit 


C Is 1,0 for a perfectly circular region* and Is larger 
for o nor* elongated region* 

The ecc on tricity Is □ dimensionless quantity* 
Which remains the same ff the region si2e Is scaled op or 
down* It represents a normal E zed moment &f Inertia about 
s line thru the region center of mass perpend icy tar to 
the region plane* If can be shown that no region can 
have an eccentricity less than 1.0, and that any shape 
other than a circle has a higher eccentricity. This Is 
because a Circle has the smallest moment of Inertia for a 
given area, 

For a I by f rectangle, the cccenfrlci+y Is 
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e - TT/f * ft 

6 \ V * 

which is 1.04 7 for a square, 1.3 1 'for a 2 Sy 1 recta no I e, 
and 2.23 for a 4 by I rectangle* for a high elongation 
i, b = TTt /€. 

Note that this definition of eccentricity [5 not 
the standard eccentricity of second order curves. T hc 
eccentricity of on elliptical region of semi-a HO 5 a and b 

3 S 



which ranges from 1 to The normel definition of the 

aCoOh + rlcity of afi ellipse Is 



which ranges from 0 to I . 

More c cm d lex region statistics could be stored on 
the tree. If the n sne y second moments are stored 
separately, then the "dom Inert axis' 1 ' thru the region 
center of mass can be easily computed* This Is e line In 
the plane of the region points through which the region 
has minimum moment of Inertia* Higher moments COUId also 
be computed, although their Interpretation In terms of 
high-level shape descriptors is less clear. Mere 
complete shape descriptors, such as the results of a 
Med la I Ax Is Transform could also be used. 
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The choice of more ton pie* shape descriptors 
depends on the part i C u I a r ■■ecojr 11 i on tg 5 k& be Enq 
performed. The simple statistics Of area, Center of 
mass, end eccentricity can yield much useful Fnfornst r 0 n, 
however, e nd attention will be focused on them, it will 
be seen they are quite useful for the e rta E y s r s of 

surface properties and simple shapes. 

3,5 Sub-programs of the Image Tree System 

P r o g r a m s have been written to obtain the image 
tree of a given scene, Measurements f ron e laboratory 
scene arc read Into an array by an imace-d 1 ssector 
Camera, and a I I st - stru C ? u r e tree Eg generated. The tree 
can be printed out, showEno the parameters associated 
with each node. Pregrams also can graph against the 
threshold any region statistic stored on The nodes, along 
Some path on the tree from a branch Tfp tp the root. The 
Original Image can be displayed,, and any arbitrary region 
can be shown su per Imposed upon It. Fe-r more detail about 
these programs, see the appendix. 
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3,6 i he Tree of a Sphere 

let us consider the tree resulting from an Image 
of a sphere with a matte surface* A matte surface 
exhibits b reflectance which is fairly unfferm In ail 
direct EPhS regardless of the angle of the Incident light* 
The image of a sphere «& a circle. If we assume the 
reflectance to he COm-p lately unlforn, and consider a 
sphEre I If from the Camera position, then the Intensity 
as e function of radius r over this circle Is 

2 \/7 

Ur} ' [ I - (r/p > 3 

The re R l& the radius of the projected circle, end the 
Intensity Is norma I lied to I at the centra] point. This 
tormy la 5 Imply expresses the fact that the projection of 
a surface scon by a views' - is proportional to the cosine 
of the angle of the viewer from the normal to the surface 
< £0 O figure 3,4)* Thus, assuming uniform scattering, the 
intensity of the fight *5 proportional to t he COE In e O f 
the Incident fand viewing) angle, The intensity value 
actually reed from the v Id 1 sec tor is t ■ C + 32 to# ( X) , 
where C Is the reading at the central point, I is the 
Intensity, and the Log Is base 2. Solving for the region 
area es a function of tho threshold t, we get 
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F1 gLiscc 3*4; Formula for the Reflectance of a Sphere 


The sphere is lit from the camera position. 









3 * 


tt-c)yr a 

A * E£ 1 - 2 J j 

where B is the ares o i the full c ire I e> He Image tree 
Should have only a single Straight branch, whose tip 
Corresponds to the Central point* Each o f the nodes on 
this branch represents a circular region centered about 
this point. 

A picture of a white sphere on a black background 
was actually read Into the computer from tnc v i d 1 sector # 
end a tree was generated by the procedure previously 
described, The tree had essentially one na I n branch, 
although there were a few very short sub-branchcs 
representing regions of very smell area, which were 
neglected. The r, ensured region area and the theoretical 
Curve are plotted together In figure 3*5* 

Note that the measured curve rises considerably 
above the theoretical curve In the central region. This 
implies t hot the intensity is not ! in ear in cosine of the 
Incident angle, but Is somewhat convex,, as in figure 3*6. 
The Sphere used for these studies had an extremely matte 
surface, and hence a negligible highlight. The sudden 
rise at the end of the curve Is due +o the threshold 
lowering to below the Intensity of points In the black 
bac kground , 
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Figure 3-.5: Region Growth for a Sphere 
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Figure 1 .6; 


Actual arid Assumed 


Surface I’sf le eta nee 
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3,7 Effect of the Specular Component 

As mu s d jscussEjd In chapter the reflectance □ + 
a surface? can he considered to be a superposition of a 
specular and a matte component* a mirrored sphere would 
give rise to a pure specular reflection, wh ic h would 
clearly be an imago of the light source, plus a 
reflection of anything el se In the room* If the surface 

is not highly mirrored, this specular component wi 11 be 
greatly at t enua t ed , so that ]t can be hftrjl acted, except 
for the Image Of the bright light source, which will bo 
sign If ic & nt despite the attenuation. This reflection of 
the light source is called & "highlight", and will 
general ly be considerably brighter than the surround Trig 
points. The magnitude of this hlghl ight relative to Ihe 
ma+tc component Is a measure of the spe-cu Iar 1 ty of the 
surface. 

Consider the effect of this high light on the 
image tree, assunine the ! Ight to come from a small 
f near I y point) source, This will produce a srra 11 , bright 
spot on top of the local maximum In the matte component, 
As a result,. a long section of the tip of the tree will 
represent a small region of fairly constant area. This 
is a result of the "spike 11, in the light intensify 


3d 


function resulting front: the small, bright highlight. 

Consider the set Of spheres shown fn figure 3*7, 
They were all painted with a mette white painty and then 
coal ed with zero through seven coats of clear enamel, 
giving them varying degrees of specularity, A graph of 
the region area vs, threshold (figure 3, S) shows the 
small flat section of tho curve representing the 
tiEgh light, for one of the spheres, Figure 3,9 gives this 
highlight depth h SS a function of the number of COats of 
I a qver, Illustrating how thu Surface specularity can be 
measured r n a slmole manner. The irregularities In this 
Curve are probably due to tho difficulty In applying the 
coats of laquer uniformly* 

3,8 The Surface Convolution 

Locally, consider a curved surface to be a part 
of a sphere of the sane radius of curvature. According 
to classical optics, a spherical mirror he s a focal 
length of one half its radius^, and will torn a virtual 
Image of the Mght source as shown in figure 3,10, If a 
light of diameter tf end distance L from the object is not 
too tar Off the Ca liter s-o b j oc t ex Is, then the dipmeter of 
its Image Is a bo u t 
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Figure 3,7; Specularity Test Spheres 
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Figure 3.8; Illustration of Depth 
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Graph is £of sphere ? of, figure 3.7 
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Figure 3,9; Highlight D<>pl;h vs. H unt O r of Laqu£r Coats 
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Figure 3. ID; The Vixt^L Icwge Made by a Spherical Mirror 
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= * ( 1 ) 
y + 2 L/ R/ 

R-r d 1 —► d * as is 3 rid eed the ca se for a f I b t 

mirror,) thus it the Sire of the light source and the 
approximate distance of the object trgir. th* camera gre 
known* the curvature e-t the surface can be determined 

near a highlight* Even It the size of the light source 

Is not known* this method gives the relative curvatures 
■f there are several different highlights In the scene, 

A good wsv to oetermfne the size of the source Is to take 

advantage of vertical f t y by knowing the approx I ma * e 

curvature of Some object In the Image* 

Meny surfaces Will "smear out" the image of the 
light, resulting En a broader highlight then would be 
g Ot t eh f r on a mfrrorcd surface of eou F V6 I ent Curvature^ 
The highlight seen can be considered to be the 
convolution of the image of the light source Bid the 
"Impulse response 1 " of the Surface reflectance, If the 
light source IS ft Sufficiently shall point, then Its 
■ mage car be considered to be an Impulse* and the surface 
'’suiear" function can be read directly from the region 
area vs. threshold curve. 



3*9 Texture 


'Texture 1 ' refers -to variations In the tight 
Intensity which arc? vary small In size compared to the 
objects being r oc can I zed . It has two basic causes* 
■"Visual texture" Is due to variations In the reflectance 
Of the surface, and tactile texture 1 " Is due to minute 
protrusions or depressions Super Imposed upon a b a S 1 C 3 : I y 
smooth surface (the serf of texture one can fee with a 
finger}. If the size of the texture Is sma I I or than the 
resolution with which the Image has been sampled, the 
Intensity variations will average out, and the texture 
will have little effect oh the tree, esldo from affecting 
the surface "smear 11 function* If the texture Is large 
enough to be d Iscemabl c-, however, It will produce e 
distinctive effect on the tree* 

Texture is a in U I T I *d i me n g I □ na I feature* and there 
□ re a correspond Inly large number of textural properties 
which could be measured* Ke are not concerned here with 
producing a complete description of texture, but rather 
with detecting features which might be useful In naming 
an object 1dontIf1CB11 on * Although such feet urns can 
help d tscr Im Inuto between objects, they do not give 
enough information to re-construct the texture exactly* 


3,9,1 Visual Texture 


Consider the two spheres Shown In figure J ( f | , 

The spheres were pa rnted With a rrptte white paint, then 
Barked with red Ink to produce visual texture, The s&he 
two spheres are shown In red, white, enc preen light* 
Since the red ink 3s highly reflective Eh the red, and 
very absorptive in the green, those lighting conditions 
produce light, medium, and heavy texture contrast 
respectively, with all other factors bo i tip held constant. 
There are two kinds of texture, with rospect to 
effect ert the Image tree. The right sphere shows small 
disconnected light patches on a connected dark 
b ac k.g round, and the left sphere shows disconnected dark 
speckles oh a Connected light background. A light spot, 
being a local maximum in the light intensity, will 
produce a tree branch. The nodes on this branch will 
represent regions the sice of the spot, and so will have 
very sms I l area, The length of the Sjronch will depend Oh 
the relative brightness Of 1 he snot compared to Its 
neighbors, since when the threshold reaches the Intensity 
Of the neighbors, the region corresponding to the spot 
will be swallowed up by the larger region surrounding it. 


Figure 3. II 


Texture Test Spheres 
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Light speckles wi II thus produce a I a r cc nunbor of sub- 
bra n c h e s whose length represents The intensity of the 
speckle, an<j whose "size" (the size of the corresponding 
regions) represents the size of the speckles. The tree 
correspond log to the 1 i g (it- spec k 1 e<J sphere p hot ogr a phed 
1 n the- green light (deepest texture) Is shown Tn 
figure 3*12* Note the Piny branches p r od u c ed by the 
sp sc k 1 e s * 

The number and length o t the Sub-branches 
prov Edes a measure of the degree of contrast of the 
texture* These euantlties are shown in figure 3*13 for 
the light- speckled sphere under the three lighting 
conditions Note how these quantities thus provide an 
Index of texture contrast. Just as the highlight depth 
and surface smear function provide an rod ex of 
specularity, I nf ermat ion about the details of the 

texture can also be obtained, up to- the limits imposed by 
the particular shape descriptors used on the nodes of the 
tree. Round speckles wl I I produce regions of low 
eccentricity, whereas streaks will produce regions of 
very high eccentricity. If the direction of The dominant 
axis of the region were recorded (corresponding to 
recording the second moments in the x end y directions 
separately), the dominant axis of the streaked texture 
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Figure 3- 12 i Tree of the Li ght-spec Vied Texture Test Sphere 

(green light) 


AH sy.h-branch nodes represent reglone of enisl. 1 area 
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Figure 3-13:: Number and Average Dnpth of Sub-branches for the 

Light-SpeckLed TeKturu Teat Spheres 
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■could tig d gT orm 1 n ed □ s we ft , 

Dark speckles will h^v e ^ different e f f cc + , 
however* Since they arc local minima in the In-tensity 
tu no t ion, rather than 1 ocs 1 max rma , they will not produce 
branches gn the tree, but rather will produce holes En 
regions, This is shewn by the tree of the da r fc-sp e c fc F ed 
sphere* shown 1 h figure 3-*H* The only effect of these 
small holes Is to raise the eccentricity of the growing 
region* as shown In figure 3 * 15* which shows the main 
branch eccentricity v? t region area for the dark~spcckletf 
sphere In the three different colored lights* Since the 
eccentricity change Is so snail, these three curves can 
be compared In this way only because all factors except 
the degree cf texture were herd a bs-o l y to l y constant - the 
same sphere was viewed from exactly the same camera 
position and with OXBCfly the same light source* Nothing 
was moved; only the filter over the light was Changed* 
The difference between the trees for the derSt 
Speckled and the light speckled spheres (figures 3 * f 2 and 
3.14) exposes 3 basic asymmetry In the Image tree with 
respect +c light and dark. This asymmetry is not just 
confined to texture* of course* Locally bright areas 
will always produce regions and hence tree nodes, while 
I oca My dark areas will always produce holes In regions. 
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Figure 3, Hi: Tree of the Dark-speckled Texture Teat Sphere 

igreen ltgJit) 
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Figure 3.15; Eccentricity ve. Region Ares for the 

Dark-speckled Texture Test SpLi^r^s 


Eccentricity 


c 







altering the Statistics of nodes that would oiherw I E)C 
exist Snym y, 

I he tree COe Id easily be extended to find dark 
specktes by generating an inverted 1 ' free for the area 
inside each region. An Inverted tree I5 3 tree In which 
the regions represent image areas less than threshold* 
instead Of greater than or e-quo! to. This will he 
further discussed In section '5 . i . 2 + 

3*9,2 Tactile Texture 

Small bump s on the surface of an object 
essentially produce many tiny "micro-objects 1 ' wfth the 
sane surface properties, If the 5 fzc of these Is below 
the resolution of the Image sampling, thE effect wIM bo 
only on fhe surface smear function, if the texture is 
larger than that, and the surface Is fairly Specular* the 
result win be many tiny highlights* producing the 
equivalent of a l 1 ght-speckIed visual texture* 

3,10 Shape 

The Image tree carries shape Information in two 
ways: In Its torn* and In the behavior of the region 
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statistics stored along Its branches, The hi + crp retat Ton 
in terms of object 5 nape of the simple region statistics 
discussed so far depends upon the object being simply 
shaped* since tho eccentricity does net give enough 
[nfprnation to distinguish between different complex- 
ShSpCd regions. Nevertheless, much U SC fu r shape 
information can be obtained even with very simple 
statistics, particularly 3ri S reccgn [t ion-or le nted 
application In which there can be restrictions on the 
shapes considered, 

3,10,1 T ho Main Branch 

Consider the object shown In figure 3,16, l+s 
tree Ts a single main branch, Just ss in the case of a 
sphere (a crude contour map F s shown In figure 3,171. 

7he simplest indicator of Its shape Is the eccentricity 
of the entire object, which Is about 1.4, clearly 
Indicating it to be quite elongated. The entire curve of 
eccentricity vs* threshold Is shown In figure 5* lb. The 
flatness of this curve Indicates that the region probably 
doesn't change Its shape very much as It grows* and that 
it has a smooth surface with no significant 
Irregularities, This Fs not a unlgue Interpretation of 
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Figure 3.16: A Hatte-wh l Cc Palntt-cl Squash 





Figure 3.17; Contour flap the Sciu^h 
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Figure 3,L&; Eccentricity Curve of the S-quasb 
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the curve f but Is a reasonable Inference given the 
assumption that the object Is net highly irregular. The 
bump j r* the ucCChtr i C fty Curve at the bright and \ S 
typical of a small newly d eve Io o1 no reg ion. Since the 
slope of the Ugh+ intensity function is very small near 
a local maximum, a smell region about that point will 
tend tp have jagged edges, pad hence a h igh epc^fi+r 1c ity. 
As the region expands, the Intensity gradient at the edge 
Increases, so the edge becomes st r a 1g hter, and the 
eccentricity Is reduced* 

Consider the plot of added region Brea, shown In 
f Igure 5*19* This quantity shows the excess area added 
to a region above the Sum of the areas of 11 S Sub™ 
regions. Since the Intensity measured is a monotonic 
function of the angre of the surface to the camera, the 
added region area is the projected area c* t hn t part of 
the surface on the object with a particular slope. A 
bump In this curve represents a large area of relatively 
low curvature* The only one Fn this case fs near the 
h t g h I i g h t * 

Figure 3,20 shows what the area added to a region 
looks like - it Is the area of a region minus tho area of 
all Ite sub^regions. Note that the statistics used arc 
Such that from the statistics of a region A and those of 
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Figure 3.19; Added Hegion Ar£ii Curve cf thv E-quaah 


Arl d etj 
region 
.■Jr&3 

A A 

(in points) 














60 


Flgurt- ^i. 20: Lilustration o£ an Added Area fcegLon; 


Shaded, area is region A-B 
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a su b-reg r 0 n B p the stat f s t Ics of the difference A-B can 
be computed, (To compute the eccentricity of region A-B, 
the ecc enfrIcit y p region ares, and center of *iass 
positron of regions A End B must all be kndwnj 
Computing Information shout the shape of such a 
difference region gives Information about bulges 
developing |n a region,, cf Erection of notion of the center 
Of mass, end other properties of all those points. C-n the 
surface within some given range of inclination to the 
c & m er e * 

The added area curve would have two peaks tor the 
hypothetical object shown In figure 5.21, due to the low 
Curvature of the annular region Indicated', In this Case 
the eccentric Tty would be constant at 1,0 jntf the Center 
ot hsss po5 It Ion woe Id be 5 + atlonary, s lr.ee the reolcns 
would alt bo Concentric circles due to the rotational 
symmetry, For the pear-like object In figure 3.22, the 
protrusion would also Increase the added area curve, but 
In this case, the eccentricity would increase as well, 
and the Center ©f nr ass would shift. 
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Figure 3,2Li A Symmetrical Object with Two Added Area Peaks 
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Figure j.iij A Contour Map of a Hypothetical O&jetfc with a 

Protrusion 



11 protr us to LI 1 " 
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3.10,2 Sub-branches 


Protrusion? of the sort 1 I lU S+ Tfi t ftd irt 
figure 3,22 w 11 P often produce s F g n I f leant sub-brarchos 
on The tree. The meaning of a sub-branch must be 
interpreted In conjunction with the Information o+ored on 
it* and on the main branch to which |t attaches. The 
attachment of a protrusion region, for example, will 
generally produce a rise In the eccen.tr It 1 fy of the rra i n 
region, and a shift In Its center of mass. The possible 
1 nterprc + B+Ions of a sub-branch depend very heavily on 
the particular Identification fqr which the tree is being 
used, A discussion of t he r n ter o r eta t I on of shape 
information for a particular set of test objects will be 
given In section 4.2. 

3,10,3 Non-interference of Texture with Shape 

Figure 3,23 shows graphs of the region area for 
the speckled s p hares of figure 3*1 I, normal T zed +o the 
light Intensity, These graphs Illustrate that the basic 
shape-describing parameters ere not affected by object 
texture In a significant way, This Is basically due to 
the averaging nature of the region descriptors used. 


Figure J,23| 
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Kefti-gti Area Curves fur the Li&ht-Speckled 
Texture Test Spheres 
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T h [ s 1 n se r\ 5 H 1 v i + y to textural r nte r f e r on ce F s a great 
improvement over most previous methods used on curved 
object's, sue 0 as Horn’s analytical met bod „ which is 
completely useless in the presence o( texture, Ldge- 
flndlog- method 5 are also contused by s ha rp texture. This 
advantage is very Important in the recognttion of real 
objects* e s w t I E be seen on the next chapter* 
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Chapter d U so on R ea l Object 5 


4 „ | Pruning 

Regions generated by smooth objects W i t h smooth 
surfaces Should In theory el ways have Smooth boundaries. 
In an actual imago, however,, minute Surface fluctuations 
and noise will cause the edge of the region to be highly 
irregular. If the Fr r egu lar i t ie s are great enough, small 
sections of the region will be detached; that Is* they 
will actually form separate small rag Ions, Since the 
area separating these s m b ! I regions from the edge of the 
nearby large region Is only slightly dimmer than the 
region points* these smell regions will join the main 
region at a threshold only slightly lower than that at 
which they started. They will thus produce very short 
branches on the image tree, whose regions are of smell 
area* These regions are essentially artifacts of the 
particular levels at which the threshold Is placed* and 
thus have ho particular significance* In order to avoid 
the waste of spa ce and t r me needed to store and analyse 
these branches* they can be h pruned 1 * away as the tree Is 
generated. This Is done s im.pty by removing! branches 
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w h l C h ara 5 hoi" + er than r f Fxed length In Intensity unit; 
end which a I sc represent reciufls smaller En area then a 
fixed size* 

Texture also produces short, small tranches f so 
thcs-a t h re s ho I d s must be adjusted 50 as no+ to threw away 
too much. The goal of pruning Is 1o eliminate all the 
really t E n y three hold-artifact branches without 
el F[ft 1 na 1 1 ng branches which represent tox+ure■ Since the 
artifact branches will be pnly one ?r two threshold 
levels deep, this car be accomplished by adjusting the 
spacing of the levels so that one level represents a 
smaller Increment of intensity then the minimal Intensity 
texture to be considered^ 

4,2 Some Sample Frees 

Various types of fruit wl II be used as executes 
of fairly simple real objects with smooth Iy Curved 
surfaces, In order to get a feel for how the Image tree 
behaves on Interesting Images* Two sarrpFo Images will 
now be discussed to give a general idea of what the trees 
arc like, 

Consider the apple In figure 4,1+ Starting at 
the brightest node end plotting the region area down the 
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Figure Apple 





70 


tree to the root gives- figure ^,2, Th& small fist 
section of the curve from 334 to 356 Is caused Li y the 
bright high! i g ht, The region area then hen I ns to grow in 
a nsriner simitar to The sphere previously discussed. The 
growth tapers off as the region expands +0 till the 
object* Then, &t about intensify 2I6 P the region breaks 
out of the object into the background. 

The region center of ness Is shown In figure 4,3. 
It stays 0 + the highlight for a while, then shifts to the 
left towards the I T t side of the apple. it then slowly 
moves io the right towards the center of the apple, as 
the region grows out to f he apple's edges. The 
occcntr 1c I f y (figure 4.4) starts off faTrly high, but 
rapidly reduces towards circularity, nearly reaching 1.0 
when the apple has filled out. 

A portion of the tree, displaying only the region 
area p a ranster, Is shown In figure 4,5+ Although the 
tree is fairly simple, a significant region Is added on 
the the na In branch at level 266. The cantor of mass 
shows this area to be above the nain region, and It In 
fact is the br Eght ref lection from the back of the stem 
hollow on the fop of the apple, which can be seen in the 
photograph In figure 4.1. It Is very common to hove Such 
a branch on the trees of fruits with stems, and hi g ho r 


F i gu re 4, 2 : 


Region Area of Llie Apple 


Region 
ar ea 

{in point s) 



depth of highLight 
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Figure 4*3; Region Center of Mass of the Apple 


y 
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Figure 4,4< Eccentricity of the 
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ure 4,5: Tree of the Apple 
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level recoan It icn routines could take iduan1a.(ie of t|> 1 S 
fact to help find Steffi areas. 
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KJ*w consider the pear shown [ rs figure 4 . 6 . Its 
trasi shown in -figure 4 + 7, is topologically similar to 
the tree of the apple, Including a snail sub-branch with 
significant area. The graphs of the various parameters^, 
however, shown In figures 4.8, 4.5, and 4.10, reveal that 
+ h|& seb-branth nas a different interpretation than In- 
Ihe case of the apple. First, IfE center of ness shows 
It to be positioned to the left of the main regron, 
rather than directly above It* Second, at the point et 
which the t w o tranche 5 join, there is a rise In the 
eccentricity In the case of the pear, whereas there is 
not In tho case of the apple. Finally, the eccentricity 
of the apple Just before brea kthroogh Into the background 
was near 1.0, whereas the eccentricity of the pea r Is 
about I . 2 j which I 5 significantly hFgh-er. Information 3s 
also available concerning the surface properties of tho 
pear. The pear’s highlight shows 2 wider ^Impulse 
response", which Indicates that its surface, although 
somewhat shiny, is not as highly specular as the apple. 
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FiKwre 4.7; Tree of Peat 
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Figure 4.-6: Center of Ho S 3 of the Pear 


Sub-hrflush tip 
300 
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4 J Useful f eaturos for Fry|t Recognition 

He Win now attempt to M st some features which 
can be easily extracted f r on the image free, so that the 
classification of fruit may be systematized, This Hist 
Is not intended to be exhaustive. In tact, quite to the 
contrary^ |+ Is intended to show that recognition of 
fruit is possible with. only a few very simple features. 

4*3.1 A Sample Set of Fruit 

In the course of studying the Image tree method, 
a large number cf fruit were processed to study the 
effects On real images. In addition, a large number of 
fruit wore given identical o r Ot es si n g under Identical 
conditions one day in order to gather some statistics on 
the various features which can be extracted* Photograph 
of the fruit fn this sample set are sho w n [ n figure 4 . | I 
The fruit used wore Bartlett pears, Macintosh apples, 
Sweet pears, and oranges. The test Images Include five 
views each of the Bartlett pears tor- a total of 25, two 
views each of the apples (total ICO, three of the sweet 
pears Ctot*| 153, and one each of the oranges. Throe 
taped images of peaches a r e a F so included In the sample 


Figure 4.11: The Fruit In the Sample Set 
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The sweet pears The ora ages 














SS-tj although they were r ecofd Od Under different 
c irtum stances, Poaches were unavailable at the time the 
samp I £ 5 0+ wa e run* 

4.5,2 SpCCufnri+Y 


As was discussed in Section 3.7, the N i mp u [ se 
response" of the surface can be a pprox imsfe I y obtained 
from the region area VS. threshold curve e+ a branch t io . 
'rte XOU I d I Ike +0 character E z e this curve I h order to 
extract some significant -features that ere useful for 
recognition purposes. One way to do this Is shown in 
figure 4,12, At iho branch tip, the second derivative cf 
the region area curve is positive due to the Specular 
component, but negative due to the matte component, A 
straight I Tne fitted to the curve at the inflection point 
Is Shown > extended to Fntorsect the axis. The 
intersec+Io-n point is called the "matte intercept", The 
value of the curve above this intercept ts used as a 
measure of the * Idth of the surface function, as shown on 
the figure. It is on! led s, for the highlight "smeer 1K 
W i d t h , 

Another measure of the surface function Is the 
amplitude of the hlghl I g h + , also marked In the figure. 
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Figure 4. 12; Characterizing the Region Axe.i Curve 
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smear width 







This Can be measured Fn various ways, but is hare 
measured as the amplitude of + he highlight above the 
matte intercept, 

A scatter (diagram of the smear wroth s v 5 . the 
highlight amplitude ti is shown Jn f laurc 4 * 12 . Note that 
the peaches, apples,, and orange are separated very well 
by their highlight properties, but that the two types Of 
pears not onFy have similar properties, but also show a 
very high degree of variation in these parameters. This 
Is partly because their surfaces are rather Fumpy and 
U Cl even* Which disrupts the highlight C eg tofi. As Will be 
teen later, this un a v on n ess can be used to help Identify 
them. 

^.3*3 Simple Global Properties 

Two very sF.upFc properties of □ fruit Bre its 
brightness end its siie. These are both properties which 
are useful only relative to some additronai information 
not contained In the Image alone; specifically, the Fight 
intensity and the object’s distance from the camera + If 
this information is available* these two f Oa fures can 
contribute recognition information. These quantities can 
be obtained* Fn many cases* fror* ether known Objects In 




Figure 4.13; Sn-.ear Width vs. Highlight: Amplitude 

far the Sample Set 
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+ h© Erraoo* Sn the ex po r im en t described fi*xt, fhe sample 
fruit were n I ( viewed with the same light intensity a r d 
5 + the same d I st s n c e from the camera, so that the Tr 
Intensity and s I zo ere comparable. 

The brightness Of art Object E S token to be the 
Enterctpt of the straight line approximation to the matte 
component with the line of Eero region area, thus 
estimating the brightness of the surface If there were no 
highlight. The overall area is estimated by scanning up 
from f he root of the tree until the first I oca 1 IT i n i ir.u m 
in the slope of the region area curve is found. The 
region area Of this no d e Is taken as the object r 5 
projected area (see figure 4.I2>* 

ft sc a tt er diagram of t he se two quantities Is 
shown |n figure 4*14 for the sample fruit. They ere 
Clearly not very useful for distinguishing between the 
fruit in the sample set. They would be very helpful if 
very l&rge objects such as watermelons were included, 
hewe ver. 

Another optical feature which could be used Is 
oof or* which would be Very powerful for fruit. This 
feature was not studied In our ex peri merits, because the 
processing of different co I or I me ge s of the seme object 
wqu i d have added complexities a nd delays without much 


F i£ure 4,14[ Brightness vS, Overall At Pa for the Sample Set 






a flt) C<1 understand In g df the fmajie tree,, 


4. *3 . 4 ov e r a | I Shape 

Our simplest shape descriptor Is just the 
eccentricity of the entire fruit outline region, whrch Is 
shown plotted with the high! Ight depth In figure 4. 15* 
Thfs parameter alone will Identify a bona no , which has 
not Seen Included in the sample set* Note that oranges 
and apples are extremely round. 


4,3,5 S-ub-branch Types 

So far, we have used only information extracted 
from the wi a I n branch* Many pro oer ties of an object 
produce su b-branchcs* In under sta nd fno an Image we must 
figure cut what these Sub-branches represent, Some types 
of sub-branches w| l | now bo discussed, and a simple sub- 
branch classification algorithm presented. 

4.3*5. I Tactile Texture 


The oranges In the sample set supply good 
examples Of tactile texture* A close examination shows 
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Figure 4.15: Cbjret Eccentricity vs. Hi.chltg.ht Depth 

fur the Sample Set 
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their surface +o be Covered by sma I 3 hurras and ysfleys, 
SlrMTS the surface Ts 3 I SO -h i o h I y specular, this 
grainlness produces nvr i?d sma I F htphlFghts, as discussed 
Fn sect Eon 5.9,2 * These produce small shgrt tranches on 
the tree, Taje+ural branches represent regions of small 
area, and are near the tfp end of the tree. The number 
of sub-branches on a tree identified os teKtyra] by the 
classification algorithm sha | I be denoted by the 
variable T. 

4* 5,5* 2 Stems 


The Ea rt I el t pears show 1 a r pe , 1ono > Fight- 
Colored stems. The branches produced by these stems are 
easily identified by their small size and t art?e 
eccentric Ity, The number of stem branches Fs denoted 
by S* 


4*3 ,5*3 Protrusions 

A pear Fs basically a spherical shape with a 
protruding bump* These protrusions will frequently 
produce a major sots-branch on the tree, as Tn the case of 
the pear discussed In section 4.2. Such protrusions 


9 ? 


generally have a I are* ere? f end usually product a 
Significant jimp En the eccentricity of the main branch 
at the point where they join It, The number ef 
protrusions will be denoted by the letter P (usually 
0 8r I 1 , 

d , 3 a 5* 4 Stem hoi i owg 

An epptE fii-s a somewhat con lea I depression on top 
fn the spot the stem Is attached, The gte-n itself Is 
sna I I er end darter than in the case of the peer* This 
stem hollow wl ll often produce a separate branch on the 
tree, os the light reflected from the hack of the hollow 
is surrounded by darker points en the rim of the hollow* 
Furthermore, the dark stem will often bisect this region, 
producing two sub-branches. Thus a significant sub- 
branch which causes a drop tn the main branch 
eccentricity when It Joins is likely to be a stem hollow, 
and this Is reinforced It there Is another sinllar region 
nearby. The number of stem hollow re £ long is denoted by 
the letter H [usually 0, l or 2), 


4 -3., 5*5 Surface Irregularities 


There are frequently fl number of branches which 
do not fpl| into any Of the above categories. These 
Often ere bus to Irregularities in the surface Of the 
Object* These Irregularities ere larger then what r 5 
called tactile texture, but smaller than those large 
eneuoh to be ea H ed protrusions. The number of such 
branches shall he denoted by the letter Tp 

4,3*6 $uh-branch Cl as$ If feat | on 

A very simple algorithm was wr rtten to classify 
sub-trmches, It Is shown In flow chart form In figure 
4*IC>* The parameter A represents the area of the sub- 
brunch just before [t joins the mo In branch. The 
parameter 4e Is the change In the ecc en +r i c 11 y of the 
halo branch at the point where the sub-branch joins, Ao 
Is positive If the sub- branch produces an i Pc reuse In the 
eccentric f+y, and negative ]f It produces a decrease.. 

The parameter j tells where on the main branch the sub- 
branch is attached, on a scale from ft*0 (matte Intercept) 
to 1*0 (full object!* It the sub-branch joins the main 
branch in the highlight region (above t ho matte 
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Ftgurt 4. Lf>: Sub-branch Classification Algorithm 
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interfile el 1 we set j to 0* 

Th i 5 a i cii* S “t hr was hand-computEd for each of the 
fruit In the ■sample. The flow Chart Fs r ec orrm ?n de d Over 
the following very sketchy description of the algorithm; 

If the biggest region represented by the sub-branch Is 
large and Increases- the eccentricity Of the main branch 
significant! y* call ft a protrusion region P. Otherwise, 
if It Increases the eccentricity only slightly, and Is 
not too small, it Is an Irregularity T. If there is no 
change In the main branch eccentricity, or if a am* 11 
positive change Is produced by p s rr a M region, then the 
sub-branch is either texture T or an Irregularity T, 
depending on whether it Is towards the tip or the root of 
the tree* Final ly, If the region Is Izarc-e enough and 
produces a drop In the main branch eccentricity, and Is 
not at the tip of the tree. It Is a stem hollow H, ft 
small region will be accepted as a stem hollow if there 
Is another stem ho Flow nearby* 

4,3.7 Object Recognition 

The flow chert of figure d+17 produces an object 
Id Ch+ r f icat ron once the Sub- branches have been 
characterized. In essence, oranges are Identified by 
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Figure 4-J7 1 Object Identification Algorithm 
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having lots of ■texture branches -and being very round. 
Apples show stem hoi lows and are very r ou r d , a stem area 
identifies a Bert left pear Immediately. The two tyoes of 
pears are sorted out on the basis of their eccentricity, 
the number of orotru 5 Ion branches, and the number of 
irregularities* Rouno objects with essentially no 
tilghl l^hts are p&achcs. 

The f low -chart shewn correctly identified all of 
the fruit with the exception of one Bartlett pear (EPM1 
which was identified as a swoet p ear., The pert inant data 
for each of the sample fruit are S ho W 0 in figure 4 . IB , 

Cur conclusion Is that recognition of Images of 
single fruits Is relatively easy, using the imago tree. 
The Image tree allows the easy extraction of enough 
Information about surface propert | e-s, shape 
Irregu I □ r It les, and general shape, as wel l as helplno to 
spot specific characteristics such as stem hollows and 
stems, and the procedures which extract this information 
are reasonably simple, More complex routines which take 
the trouble to TO'Ok more c I ose I y at the tree’s statistics 
should be even more reliable* 

The recognition procedures described would be 
disrupted {as would many 0 1 h e r 5 I by cc c I u s 1 (ins, Shadows, 
missing stems, and object positions which hide 
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Figure 4*18: Eccentricity and Sub-brei-tch Types for the 
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S [gn Ff leant features. Karty o( these problems could -be 
eased by a suitable vertical system, which could use 
other knowledge to exp I a F n and Correct changes in the 
Smege free* Other problems can be solved without higher- 
level eld, simply by making the recognition routines more 
C t CV er , For examp f e, occlusions cars generally be 
detected by the way in which two regions connect. Once 
an Object is known to be part la My occluded, correct Ions 
Can be made to Its region statistics which n E ve an Idea 
Ot Its f crjfi^ under the assumption that the visible and 
the hidden parts art similar* 

Even In the presence of Severe occlusion 
problems, the tree still gives valuable local Information 
about highlights and texty r e. Although the stems gave 
significant gld In identifying Bartlett pears, the stems 
were Hot seen |n ten of the test cases, yet mins of those 
were correctly identified. 

4. i Feces 

This sect lem Illustrates the behavior of t ha 
image tree produced from a more complex smoothly curved 
object; a humgn face. It Is included to Shew another 
example of a real recognition task for which the Image 


■free 1% potentially useful, ft tree was generated from an 
linage of a face, seen, fell face and I It from the front* 
This tree Is shown In figure 4,19* Branches of the tree 
have base labeled with the local max Ina Oh the face to 
which they c or respond, and the shapes and positions o f 
these regions Is shown In figure 4.20, l hose regions 
might be useful fer face recognition, at least for the 
simple angle of view and lighting considered here. 

Contour maps at 3 single level of the tree are 
Shown Fn figure 4*21, for each of two levels (marked In 
figure 4*19), At level 313, most of the major regions 
seen In the photo appear, wF+h the except tor. of the lower 
lip high1 r ght, which Is consFdurably dimmer. The contour 
map at level 26S Is rather E nterast 1ng . Consider not the 
region Included within the contour, but the area 
excluded. This includes most of the mouth, the eyebrows, 
the eyelids (the eyes are closed), the nostrils, and a 
shadow area on el+hor side of the nose* These are 
locally darte areas In the image. These could be Isolated 
by making an inverted tree - that is, by making a Tree 
with the Inage negated. These locally darli ares 5 are 
probably better places to begin face location, since 
there are fewer Of them than there are locally bright 
areas, and they are more prominent. Indeed, there are 
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Figure 4.19; Ires of a Face 
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Figure 4.20: Seine o£ the Regions of the Face Tree 
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Regions corresponding to bo^sd nodes 



Ffic.e slotie 


Regions super i mpo ecd on fic« 













Figure 4. 2 If Slice* of the Tree at Two Thresholds 



Threshold 26 ® 


Threshold 313 










Gx per i me rtf £ which ; nd (cate that a s babies I era r th to -see 
fates, they first fixate on the pair of eyes L I, 6J. Once 
a face F s roughly fotited, h F g her level rout F n e 5 tan. make 
sense of the locally brroiit areas with less difficuFty, 
Figure 4.22 Show 5 a contour map Kith both levels 
Superimposed, with the dark re C Ions Shaded. 

Note that the Image tree c £ n easily be used to 
F sol a to facial features and determine their approx (mate 
position. In order to better characterise their shapes, 
more complex shape descriptors would probably bo needed 
than those whfqh have been used so *ar. The (made tree 
cam be used to characterize the shapes of objects, such 
as noses, which have mo "herd edge 1 * boundary, Th F 5 will 
be further discussed in section 5*2* A * 


Figure 4,22: The Two Contour Hap: Super imposed 
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Shaded area btfto'«.’ threshold 268 







Chapter 5- Discussion 


5-. I Comparison with Previous Work 

The 1 ma ge tree can now be situated b mong the 
pattern recognition methods discussed In chapter 2, ft 
t 5 a " r e g IO n. S 11 method, rather than e n ed q e d e t oc t I o o 
scheme, and does no d 11 ferent let I on or other pre¬ 
processing Of the image* It extract? Information ebou + 
both the surface properties of an object and about its 
shape, it does not regu Ire any high degree of precision 
of measurement with regard to the exact location of 
specific points In the image, and does not make any 
essentia f use pf perspective Information. It does not 
attack problems of the 1, parsrng ,r of an image into Its 
component parts direct iy, all-hough It may aid this 
process by the way It organ lies the image information. 

5,2 Advantages 

The image tree has a number of advantages for 
pattern recognition over many previously used met hods. 


5 * 2*1 No Edge Problems 

By Jirlue of being a "rug Ions" method, The imgge 
"Ire* Is immune to many of the problems which beset edge 
(Je^flcfors* These include false edges end gaps in reel 
ed ges* 

5 m 2 t 2 Noise Immunity 

The USO of moments a nd aver? ge S as parameters On 
the tree results In a System which does not reouire great 
precis loo of Measurement, and which Is in Sensitive to 
noise end distortion In the image. 

Obtains and Separates Surface and Shape Information 

The tree also carries t nf orrrgT ion about both 
Surface properties and shape* The two forms of 
information ere welt separated on the tree, with the 
surface Information being carried at branch tips* and the 
shape Information appearing further down the branch* 

indeed, the tree can be thought of In a general 
way as carrying detail Information near the tips of 
branches, and lower resolution Information towards the 
r-Oot* For an Image containing many objects, the root 


Win represent the entire scene, and yarrows sub-branches 
wEM represent Si?t)-parts, end then sub-parts of the SUb- 
partE, The tree qsn thus be thought of as prpvIdEng * 
renoe of measurements of d E f t c r F n g degrees of a c u 11 y * 
These not Eons of pa ttern r e c o g n i t I OP as a 50 rt O t 
"measuring 1 ' problem are due to K1 r st h , 

5.2,4 Objects Without Bounder E e s 

The Image tree E 5 easy to apply to the 
r eC ago it Eon of objects without rest edges or well-defined 
bou ndar 1 es p such as a nose, or afi object Eft so that one 
side tad 65 off gradually Into shadow. Assuming the 
object produces a separate tree branch. It can be 
ana I y2ed from the data at the tip pf the trench, wording 
down towards the be 50 until the parameters EndFc&te that 
the region Is taking Tn tpo much extraneous area to be 
useful* Thus some 1 ntorrr.at I on about a nose can be 
extracted ever though tt has no well-defined upper 
boundary, because tt has well-defined lower and side 
boundaries. This simple task can be rat ho r complicated 
for ed ga-pr tented procedures, or for programs which are 
regions orient ed but which, do not me Ike a sa r Ec s o# 
related measurements at different levels, as In the tree. 


By the same arguments, the tree will contain Information 
about a smoothly curved object even ft ft Is partially 
obscured, provided ft cqntalns a local -brightness 
maximum. The procedures which analyze the tree must be 
able to detect the occlusion and to try to condensate for 
it. 


5,3 Problems 


The separation of coarse and fin* Information Is 
not always maintained by the tree, unfortunately, When 
branches representing two different Objects merge, 
Information about those parts of the object not yet 
filled out by the region may be lost. If a small 
highlight area is sw-sl lowed up by a larger region before 
achieving much depth In Its own right, the information 
that would have been obtained about the local surface 
properties of that area are swamped ogt, When a region 
representing some object l n a scene Joins with a larger 
region representing the background, the Information about 
the smaller object Is lost. One case in which this can 
occur Is When a dark Object Is on a light background, or 
hear a lighter object. Or, alternatively, a region may 
extend beyond the boundaries Of an object on one side 


before reaching the boundary on the other side, possibly 
due to an overall gradient In the 1 ' g h t Intensity* This 
shall be referred to as a " bra o ktti r ou oh Although It 
tan usually bo es 5 11 y detected by Its -effect on. the 
region parameters {Sharp rise In the region area and 
eccentricity, and sudden shift in the center of mass), H 
still means □ loss of Information about the side of 1 he 
object Which the region has not H f|lled rl . 

5,4 Further Considerations 

5* 4, l Other Stat I st les 

So far region shape has been character tied by the 
region area, eccentricity, and canter ot mass position* 
ThorC gre many other region statistics which COU Id be 
used to characterize the regions, depending upon, the 
part leu Far recognition, task at hand, 

0-ne very simple addition which C 0-U I d be made 
kou id be to compute the x and y second moments 
separately, so that the major axis Of the region cculd be 
found* This Is the axis about which the regTon has a 
minimal moment of Inertia, This would allow the t r ee 


parsing prog ran 5 to check highly eccentric regions for 
proper or reri+g+icn, given some hypothesis about what the 
region C a presents. Other Maher moments could also be 
used, although their i nt erpretat ion would be somewhat 
more difficult* 

It would be very useful to know If a region has 
any ma j or holes ih |t, |t Is very easy to tell it ft has 
any holes at all by calculating Its cu l c r number while 
the conn set edness of the region Is being verified. This 
number can be calculated on a local basis, using 
procedures reported by Gray []T^ t Since the euler number 
gives the number of objects minus the number of holes, 
and since the reglor is known to be one connected object, 
the number of ho ' 6S in it Is one minus the euier number. 
Unfortu rarely, this Is the topological number of hp i es t 
rathe ^ than the number of holes of large sieii. Of 
course, there ran be no large holes if there are no holes 
at all, so the euler number can be used to see if a test 
for significant holes Is needed. Unfortunately, a l&rpc 
percentage of regions generated by real images will have 
small holes In IhSdft, especially near the edgB, so this 
test will not reject very many* If a region Is known to 
have no major holes, and has a high eccentricity, then ( + 
must be elongated* In the absence of a "major hole" 


prod leaf e ( there Fs ?! ways the possibility that a h i Oh 
ecc ert r lc Ffy may be due +0 a perfectly round rftalcn ( but 
vl th s large hole In the middle, 

1 n genera I , any sort of S ha pe -de SC r F pt 0 r 
5 Inor F t hm can be applied to the regions, such a £ the Blum 
algorithm (Medial Axis Transform) [jjj, I bfilFeve, 
however, that one of the strengths of the Image free ns a 
method is to allow easy r ec og h It r on with relatively 
simple region shape descr ip tors. Us Tog very complicated 
descriptors hot only Will consume a great deaf of 
computer time, but will also conpl tests the ana iy sis 
required of the higher-level programs* t\ more detailed 
shape analysis should probably be reserved for cases in 
which problems arise in the simpler procedures. 

5,3,2 Region-hole Duality 

Tb3 tree procedures are not symmetric with 
respect to light and dork, as has be on point ed out 
earlier. Thus 3 blacfc spot an a light object Is not 
perceived es ?n object, but as a hole Fn & re-gfO'n, 
furthermore, these holes are not detected by the 
programs, and Insufficient information Is stored on the 
tree to tell that they ere there. Thus the etfett of a 


h=o 1 ■& Is to decrease the region area, and Increase the 
i 1 Boion 6 c c en TC Ec i ty , but it I? net detected a $ a hole per 
set E n the detection of tenture, block speckles have a 
completely d I f f e r e n t effect than w h E te speckles* An 
obj ec t is harder to recognize on a white background than 
on b dark one* 

This f s not 3 desireable situation. Ah object 
Should be easy to recognlfc OH any highly contrasting 
background, regardless of whether it Es darker or lighter 
than the object, A possible solution would be *o make 
two trees, one wf+h the Tmage negated'. Thus one would be 
the tree already discussed In detaM, and the other would 
be a tree of dark regions on lighter ba c knre-und s i i ft 
Which the t F p 5 Of the branches would represent locally 
dark areas, rather than locally light ones* F 5 r the face 
Considered fn section, 4 + these dark branches would 
represent significant locally dark areas, such 3 S the eye 
sockets, the nostrils, and The dark areas along the side 
of +he no&e. The eye sockets and the nostrils, En 
particular, are probably very Important In orienting 
visually with respect to e face. 

There i& no reason why this pr ocedu re should not 
be carried to more than one level. Whenever a re g E on Is 
Isolated, the contiguity scan routines couEd be called 
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again, bu+ SC an Hi 3 n ^ only inside the recton, and with 
their sense inverted,,, so that they would 1 1 nd holes. 

Small holes could then be eliminated, but if there v?re 
■any largo ones, *hey wou Id be noted on the tref*. 
Furthermore, the sense could then be inverted once more, 
and the coot fgu Ity scan tried once again to find 
additional light regions Inside t ha dark holes, 

This procedure would succeed In finding a dark 
apple on a light background* The apple could be Isolated 
by an Inverted run of the tree procedures, and then the 
normal procedure could be carried ovt on the region thus 
3 ao lated, 

S*4*J Complex Lighting 

In the above discussion. It was assumed that the 
Illumination was coming from a single do T n t source. 
Changing the source of the Illumination will change the 
properties * f the highlight region, but will not alter 
the basic properties of the tree. If the Illumination Is 
from a diffuse source, specularity information Is lost* 
Light from seV'ftral point sources wl II produce multiple 
h | gh I i ght s. [f the high level parsing routines know 
about the 1 I ght source, they can compensate for these 
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effects* By making hypotheses about the Objects in The 
image, these routines ecu Id equally well find out about 
the lighting from the image, 

5.4,4 Isolations of Regions 

A by-product of the Image tree is the isolation 
of regions which can be used as data tor other feature 
extract Ion programs. One might, for example, take a 
fairly large region around the htohf ioht, subtract cut 
the sna l | region containing the hFghl Ight Itself, and 
hentf this difference region to a textural analysis 
program, this program could Use this region to extract 
texture Information in various ways, such as performing a 
Fourier transform, au + OCQnvo1Ut1 OnJ or similar 
processing, obtaining information about surface speckles 
not available directly from the tree. Using, a region 
generated from one of the tree redos helps assure that 
the portion of the image upon which the analysis is 
performed Is a suitable one. 


5 Summary and Conclusions 


A procedure has been outlined for processing 
Images of t hree-d 1 mo ns iona I objects n E fh smoothly curved 
surfaces* The method Is able to extract some information 
about the surface properties of the objects > such S3 the 
texture, specular It y., and surface Irregularity. 
Information about she pe Is a I SO extracted, The 
procedures are Insensitive to noise eng distortion, and 
can be used to perform real recognition tasks. It Is 
hoped that this work will provide a stepp 1 ng-stone in the 
challenging study of c om put er vision* 


ft p p- cn d i x ; 


0E?crrpfFon Of Algorithms 


This append 3 k contains an outline Of the 
algorithms used In the tree generating urogram. 

The Image tree Is generated one threshold level 
at & time, starting &t the highest level (branch tips), 

At each level, the Image Is scanned, and the points above 
the threshold are marked In a scratch array* This 
scratch, array rs then scanned for marked points* When one 
Is found, a contiguity routine Is called, which visits 
ell marked points which can be rsoched from the Start via 
6 connected path. The marks are erased by this routine 
as It goes, and statistics are k fl oT on the region thus 
generated, such as the sums of the x and y coordinates of 
the points, end the sum of the souares of the x end y 
coord Inates (used to compute the center of mass and the 
eccentricity), A tree node Is then made up for the 
region,* and the scon tor marked points Continues, A 
special mark Is left in the scratch array for each 
region, When this mark Is encountered during the scan 
at the ne:*t level. It Is looked up on an association 
list* This e sto b I is he s the link between a region and 
the regions which are h Subset of It at the previous 


I eve i - 1 , 0 . between a node and Its g,ub - fi od os, 

The contiguity cca n 1? the most complex program. 

E t wfirhs by leaving directional pointers In the scratch 
array^ These are three-tilt codes denoting one of the 
flight possible neighboring points. The contiguity scan 
fs always started at a point which Is o-n the bottom edge 
Of the region, it traces along this edge to the right by 
moving from one marked point to the next, but always 
keeping an un-mar ked point to the right side. As It 
goes, It erases the marks, so that tor a region with 
smooth boundaries, it will tol low a spiral path tc The 
center, "eating up 11 the marks OS It goes, like a laths 
with, the tool continually advancing Into the work* 

As the contiguity routine scans, ft lays -down 
back pointers in the scratch array which enable 1+ to 
retrace Its path hack to the start. If a dead end is 
reached I no more marked neighbors?. It traces back alone 
this path, looking for narked points to the right. There 
can be homarked points on the left side while 
backtracking, since this was the right Side on the way 
out, end the outgoing scan stoyed as far to the right as 
possible, I# a m&rked point Is found on the backtrace, 
it Fs replaced with a pointer to the adjacent path 
already traced out, and then a new path Is traced as if 


this wG r C 3 new starting. point* tihere the bncMra ce 
reacsics the original start i no pointy the cent [gu ity scan 
i s c op plated* the cffocl of this □ I gor f f hm Fs to 
const rye t a trot Of pointers In the scratch array, with 
the starting point at the roof, All points wM ch can be 
reached via a connected path from the starting point will 
be a part of this tree, am example of which [5 shown in 
tIgu r» A1 4 

An a I g or I + h m developed by £ „ Bryafi OOuld 

speed the contiguity scan considerably. It entails 
cod trig tha scratch array line by line as strips, os In 
figure hi , Each strip fs specified. by its y coordinate* 
and the x cnpnj ins te? of Fts left and right end. The 
contiguity of these strips Is than checked, rather than 
operating on the individual points. This algorithm not 
■only avoids scanning the entire scratch array, most of 
which is blank* but also reou ires fewer operations to 
find 1 all of the contiguous, points, since they ar * 
gathered Into croups. It thus takes advantage of the 
fact that regions produced by real Images, as opposed to 
random noise, will tend to have the points clustered into 
bynche s. 

A number of other programs were written in the 


course Of this research 


In order to make it convenient 


figure AL; The Tree yf Fointers Layed Down by the Contiguity 

Sean Algorithm 


(Sho'Wfl fof an arbitrary regionJ 


* - marked poiptj ihtluded in region 

V= pointer in direction of rout 

(arrowhead not shown due to small si^e) 




Root of Ere£ 










Figure A2; A Region Coded as Strips 


The same region is used as in figure AI 
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to Study a largo: number of trass* programs were written 
to print out the trees on the Mne^printer, w F t h the 
sign |f leant parameters associated with each rode* 
Furthermore, a program was produced to plot any parameter 
vs, threshold slang any set of branches of the tree. 

This program was used to produce the graphs in thfs 
pa per . 

Programs were also written to display an 
Intensity modulated picture of the Image* using the seven 
Intensity levels of a DEC 34D display* Since our J4Q hse 
no fast raster .mode, a display comp I(er was written Which 
generates 0 display list In Increment mode, allowing 
fairly large Images to he shown virtually f I it ke r- f r ee, 
Other routines enable any arbitrary region in the Image 
to bo shown superimposed on this nlctyrc* The pointer 
method used |n the contiguity scan was actually written 
for these display routines, which were developed first* 
The ex 1 stance of this program made the writing of the 
Contiguity scan very simple, which is one reason why 
faster algorithms such as the liryg n algorithm were not 
soug ht, 

ft large amount of code was required to back up 
the programs mentioned above. This Includes a dynamic 
storage allocator for manipulating u forge mutter of 


arrays of changing s display 3 rd plotter routines and 

O'! Per I/O routines, rout ires tor manipulating list 
structure, aria routines which fflap arbitrary Ideal 
p f OC Cd ures oyer an array. The pro^rep'ia c om prise ov 0 r 
5200 words of PDP™!0 MIDA5 assembly language code* not 
including about 1700 words of fixed buffer and tables, 
and not including the dynamically afloeafEd array and 
list structure area* which can grow to an arbitrary size* 
Also used was the CHTOUR program [12D, which 
draws Intensity contour maps of an image, and Which was 
written early In the course of this re search, before +ho 
exact area of study had been decided upon. 
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